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This paper presents the formulation of an uncertainty quantification chal- 
lenge problem consisting of five subproblems. These problems focus on 
key aspects of uncertainty characterization, sensitivity analysis, uncertainty 
propagation, extreme-case analysis, and robust design. 


I. Introduction 

NASA missions often involve the development of new vehicles and systems that must 
be designed to operate in harsh domains with a wide array of operating conditions. These 
missions involve high-consequence and safety-critical systems for which quantitative data 
is either very sparse or prohibitively expensive to collect. Limited heritage data may ex- 
ist, but is also usually sparse and may not be directly applicable to the system of interest, 
making uncertainty quantification extremely challenging. NASA modeling and simulation 
standards require estimates of uncertainty and descriptions of any processes used to ob- 
tain these estimates. The NASA Langley Research Center has developed an uncertainty 
quantification challenge problem in an effort to focus a community of researchers towards a 
common problem. This challenge problem features key issues in both uncertainty quantifi- 
cation and robust design using a discipline-independent formulation. While the formulation 
is indeed discipline-independent, the underlying model, as well as the requirements imposed 
upon it, describes a realistic aeronautics application. A few high-level details of this ap- 
plication are provided at the end of this document. Additional information is available at: 
http://uqtools.larc.nasa.gov/nda-uq-challenge-problem-2014/. 

II. Uncertainty Classification 

This challenge problem adopted the generally accepted classifications of uncertainty re- 
ferred to as aleatory and epistemic [1, 2]. Aleatory uncertainty (also called irreducible 
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uncertainty, stochastic uncertainty, or variability) is uncertainty due to inherent variation 
or randomness. Epistemic uncertainty is uncertainty that arises due to a lack of knowledge. 
Epistemic uncertainty is not an inherent property of the system, but instead it represents 
the state of knowledge of the parameter and as such it may be reduced if more information 
is acquired. 

According to its physical origin and the system’s operating conditions, the value of a pa- 
rameter can be either fixed (e.g., the mass of a specific element produced by a manufacturing 
process) or varying (e.g., the mass of any element that can be produced by a manufacturing 
process). The physical origin of a parameter as well as the knowledge we have about it 
must be used to create uncertainty models for it. Intervals, fuzzy sets, random variables, 
probability boxes [3], a.k.a. p-boxes, etc., are different classes of uncertainty models. 

While a parameter may be known to be aleatory, sufficient data may not be available to 
adequately model it as a single random variable. In this case, an approach is to use a random 
variable with a fixed functional form, e.g., a normal variable but the specific parameters 
required to fully prescribe it, e.g., the mean and standard deviation, are unknown constants 
assumed to lie in some given bounded intervals. This results in a distributional p-box , where 
the physical parameter is indeed an aleatory uncertainty but the parameters prescribing its 
mathematical model are epistemic uncertainties. A distributional p-box prescribes all the 
elements of a family of random variables. Conversely, a free p-box is defined by prescribed 
upper and lower bounding cumulative distribution functions and admits any random variable 
whose cumulative distribution function falls between these bounding functions. 

The above considerations lead us to classify each uncertain parameter of the challenge 
problem as belonging to one the following three categories: 

I) An aleatory uncertainty modeled as a random variable with a fixed functional form 
and known coefficients. This mathematical model is irreducible. 

II) An epistemic uncertainty modeled as a fixed but unknown constant that lies within a 
given interval. This interval is reducible. 

Ill) An aleatory uncertainty modeled as a distributional p-box. Each of the parameters 
prescribing the random variable is an unknown element of a given interval. These 
intervals are reducible. 

Note that there is no epistemic space associated with a category I parameter since its 
uncertainty model is fully prescribed. The epistemic space of a category II parameter, which 
belongs to a family of infinitely many deterministic values, is an interval. The epistemic space 
of a category III parameter, which belongs to a family of infinitely many probabilistic models, 
is the Cartesian product of the intervals associated with all the epistemically uncertain 
parameters of the random variable. Category III parameters may stem from a case where a 
parameter is known to be random, but there is insufficient data to determine the parameters 
of its distribution. In this case the epistemic space is given by the confidence intervals of 
such parameters. 

Since most models, especially those used to describe uncertainty, are imperfect; the 
possibility of improving them always exists. Specifics on what we mean by an “improved” 
or “reduced” uncertainty model are now in order. Improvements over any given uncertainty 
model are attained when its epistemic space is reduced. This reduction can be attained by 
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performing additional experiments or doing better computational simulations. For instance, 
denote by a distributional p-box with a Gaussian functional form, mean /i e [a, b\ and 
standard deviation o e [c,d\. The uncertainty model A "! 2 is an improvement over Afi if its 
epistemic space e 2 satisfies e 2 C [a, b } x [c, d\. Conversely, an irreducible model remains fixed 
throughout the uncertainty quantification process. 

III. Problem Formulation 

Let S denote the mathematical model of the multidisciplinary system under investiga- 
tion. This model is used to evaluate the performance of a physical system and evaluate its 
suitability. Denote by p a vector of parameters in the system model whose value is uncertain 
and by d a vector of design variables whose value can be set by the analyst. Furthermore, 
denote by g a set of requirement metrics used to evaluate the system’s performance. The 
value of g depends on both p and d. The system will be regarded as requirement compliant if 
it satisfies the set of inequality * 3, constraints g < 0 . For a fixed value of the design variable d , 
the set of p-points where g < 0 is called the safe domain , while its complement set is called 
the failure domain. Therefore, the failure domain corresponding to a fixed design point is 
comprised of all the parameter points p where at least one of the requirements is violated. 

The relationship between the inputs p and d, and the output g is given by several 
functions, each representing a different subsystem or discipline. The function prescribing 
the output of the multidisciplinary system is given by 

g = f(x,d), ( 1 ) 

where a? is a set of intermediate variables whose dependence on p is given by 

x = h(p). ( 2 ) 

The components of x , which can be interpreted as outputs of the fixed discipline analyses 
in (2), are the inputs to the cross-discipline analyses in (1). The components of g and x are 
continuous functions of the inputs that prescribe them. 

Challenge Overview 

In the following subproblems, initial uncertainty models for the uncertain parameters in 
vector p, as well as software to evaluate Equations (1) and (2), are given (see section V 
for software availability). We also provide some data, hereafter referred to as “experimental 
data” , which functions as a surrogate for experimental results. These data are exempt from 
model form uncertainty. An overview of the various tasks of interest is as follows. 

• Improve the initial uncertainty models based on the experimental data. 

• Decide which uncertainty models should be improved such that the spread in various 
quantities dependent on p is reduced the most. 

• Determine whether various quantities dependent on p are sufficiently insensitive to the 
uncertainty in any given parameter such that the parameter can be assumed to take 
on a fixed constant value. 

a Vector inequalities apply to all vector’s components. 
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• Determine the range of selected metrics that depend on p for two uncertainty models of 
p. The first uncertainty model is based on information provided in this document. The 
second model, which is obtained through interaction between the respondents and the 
challenge problem hosts, is a refinement of the first model. The first and second models 
differ in the description of a subset of the parameters to be chosen by the respondents. 
These improved models are based on further experimentation and observation of p. 

• Identify the particular uncertainty models that yield the extreme values of the ranges 
mentioned above. 

• Determine the design points that provide optimal worst-case probabilistic performance 
in the presence of uncertainty (optional). 

These tasks are at the core of the subproblems presented below. The subproblems, 
namely uncertainty characterization, sensitivity analysis, uncertainty propagation, extreme- 
case analysis and robust design, are presented in sections A, B, C, D, and E respectively. 
While the subproblems A, B, C, and D pose analysis tasks for which the value of d is kept 
fixed at d = dbaseimej subproblem E poses design tasks for which the value of d is to be 
determined. Specifics on each the subproblems are provided next. 

Subproblem A: Uncertainty Characterization 

Here we consider a subsystem of S whose scalar output X\ depends on five uncertain 
parameters 13 as given by 


xi = h 1 (p 1 ,p 2 ,p 3 ,p 4 ,p 5 ). (3) 

Specific information on these parameters is provided in Table 1. The first column provides 
the parameter’s symbol, the second one its category (see above for a description of the 
categories), and the third one describes its uncertainty model c . While the symbol A denotes 
the support set or parameter range, p, E[-], H[-], and P[-] denote the correlation, expected 
value, variance, and probability operators respectively. In this subproblem, the tasks of 


Table 1. Uncertain parameters. 


Symbol 

Category 

Uncertainty Model 

Pi 

III 

Unimodal Beta, 3/5 < E[ Pl \ < 4/5, 1/50 < V[ Pl ] < 1/25 

P2 

II 

A = [0,1] 

P3 

I 

Uniform, A = [0, 1] 

P4 5 P5 

III 

Normal, -5 < E[ Pi ] < 5, 1/400 < V[ Pi ] < 4, \p\ < 1 for i = 4, 5 


interests are as follows: 

b The components of vector quantities and vector functions will be specified as subscripts, e.g., the scalar 
p x is the first component of p while hi is the first function of h. 

C A random variable whose probability density function (PDF) has a single peak at the interior of the 
support set will be called unimodal. 
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Al) Software to evaluate hi and n — 25 observations of xi corresponding to the “true 
uncertainty model” are provided, i.e., a model where Pi is a fully prescribed Beta 
random variable, p 2 is a fixed constant and p A and p 5 are described by a single and 
possibly correlated bivariate normal random variable. Use this information to improve 
the uncertainty model of the category II and III parameters (refer to Section II for 
the definition of a reduced/improved uncertainty model). The resulting models should 
only exclude the elements of the original models that fail to explain the observations. 

A2) Use an additional n = 25 observations to validate the models found in Al. 

A3) Improve the uncertainty models further by using all the 50 samples available. 

A4) Account for the effect of the number of observations n on the fidelity of the resulting 
uncertainty models. How much better is the model found in A3 as compared to the 
model found in Al? 

Subproblem B: Sensitivity Analysis 

We now consider the multidisciplinary system S having the input p G M 21 and the output 
g G M 8 . The first 5 input parameters should be modeled using the results from task A3, 
while the remaining 16 parameters are given in Table 2. 


Table 2. Uncertain parameters. 


Symbol 

Category 

Uncertainty Model 

PG 

II 

A = [0,1] 

P7 

III 

Beta, 0.982 < a < 3.537, 0.619 < b < 1.080 

P8 

III 

Beta, 7.450 < a < 14.093, 4.285 < b < 7.864 

P9 

I 

Uniform, A = [0, 1] 

PlO 

III 

Beta, 1.520 < a < 4.513, 1.536 < b < 4.750 

Pll 

I 

Uniform, A = [0, 1] 

Pl2 

II 

A = [0,1] 

Pl3 

III 

Beta, 0.412 < a < 0.737, 1.000 < b < 2.068 

Pl4 

III 

Beta, 0.931 < a < 2.169, 1.000 < b < 2.407 

Pl5 

III 

Beta, 5.435 < a < 7.095, 5.287 < b < 6.945 

Pl6 

II 

A = [0,1] 

P17 

III 

Beta, 1.060 < a < 1.662, 1.000 < b < 1.488 

Pl8 

III 

Beta, 1.000 < a < 4.266, 0.553 < b < 1.000 

Pl9 

I 

Uniform, A = [0, 1] 

P20 

III 

Beta, 7.530 < a < 13.492, 4.711 < b < 8.148 

P21 

III 

Beta, 0.421 < a < 1.000, 7.772 < b < 29.621 


The relationship between the input p and the output g is given by Equations (1) and 
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(2), where the intermediate variable x G M 5 is given by (3) and 


X 2 = h 2 (p 6 ,p 7 ,p 8 ,p 9 ,p 10 ), 

(4) 

*3 = fr 3 (Pll,Pl2,Pl3,Pl4>Pl5)> 

(5) 

x 4 = h 4 (p w , p 17 , p 18 , p ig , p 20 ) , 

(6) 

X 5 =P 21. 

(7) 


Note that the propagation of the uncertainty model of p through h yields distributional 
p-boxes for x 4 , .xq. x : > and x 4 . If the uncertainty models of the category II and III parameters 
are improved, so will be the resulting p-boxes of x 4 , X 2 , x 8 and x 4 . 

In this subproblem we want to perform the following tasks: 

Bl) Rank the 4 category II-III parameters entering Equation (3) according to degree of 
refinement in the p-box of x 4 which one could hope to obtain by refining their un- 
certainty models. Are there any parameters that can be assumed to take on a fixed 
constant value without incurring a significant error? If so, evaluate/bound this error, 
list which parameters and set their corresponding constant values. Do the same for 
the 4 category II-III parameters prescribing X 2 , x->, and x 4 . 

B2) Rank the 17 category II-III parameters of Tables 1 and 2 according to the reduction 
in the range of the expected value 

J\ E[w(p, dbaseline)], (8) 

which one could hope to obtain by refining their uncertainty models. In this expression, 

w(p, d ) = max o,. = max fi(h(p), d), (9) 

1<£<8 

is the worst-case requirement metric. Are there any parameters that can be assumed 
to take on a fixed constant value without incurring a significant error? If so, eval- 
uate/bound this error, list which parameters and set their corresponding constant 
values. 

B3) Rank the 17 category II-III parameters of Tables 1 and 2 according to the reduction 
in the range of the failure probability: 

J 2 = 1 P [tc(j P, ^baseline) 0] , (10) 

which one could hope to obtain by refining their uncertainty models d . Are there any 
parameters that can be assumed to take on a fixed constant value without incurring 
significant error? If so, evaluate/bound this error, list which parameters and set their 
corresponding constant values. 


Compare the above rankings and eventual parameter simplifications. Note that while the 
tasks in Bl are of interest to experts in the disciplines modelled by h, the tasks in B2 and 
B3 are of interest to analysts of the integrated system. Further notice that each ranking can 
be used to determine the key parameters whose uncertainty model we want to improve. 


d Note that J 2 is equal to P 


ULliP 1 ^baseline) > 0} . 
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Subproblem C: Uncertainty Propagation 

This subproblem aims at finding the ranges of the metrics J\ and J 2 in Equations (8) 
and (10) that result from propagating both the original uncertainty model and a reduced 
one. The reduced model, in which the respondents chose 4 out of the 17 category If and 111 
parameters to be improved, was provided by NASA. This is a practical limitation that may 
stem from having a limited amount of time or money to generate better uncertainty models. 
In particular, the tasks of interest are as follows: 

Cl) Find the range of J\ corresponding to an uncertainty model based on your response to 
A3 and the information in Table 2. 

C2) Find the range of J 2 corresponding to an uncertainty model based on your response to 
A3 and the information in Table 2. 

C3) Select 4 category II-III parameters out of the 17 available according to the rankings 
in B2 and B3, and request from us an improved uncertainty model for them. While 
improved models for any four parameters will likely lead to smaller ranges of variation, 
the set of 4 leading to the smallest ranges is ideal. Each working group may request 
a reduced uncertainty model of 4 parameters of their choice. Only one set of reduced 
parameters will be provided to each working group. 

C4) Use the reduced uncertainty model to recalculate the ranges of J\ and J 2 . 

A cautionary note on the approaches used to calculate the ranges of J\ and J 2 is in 
order. Methods to calculate these ranges may lead to underpredictions or overpredictions of 
the actual range. Each of these two outcomes has its own drawbacks. An underprediction 
(e.g., a situation where the search for the end points of the range fails to converge to a global 
optima) can lead the decision maker to the wrong decision (e.g., the estimate of largest failure 
probability is half of the actual value). An overprediction (e.g., a situation resulting from 
replacing a distributional p-box with a free p-box) can not only lead the decision maker to 
the wrong decision (e.g., the estimate of largest failure probability is twice the actual value) 
but also prevent him/her from making any decision (e.g., the predicted range of failure 
probabilities covers the entire [0, 1] interval). 

Subproblem D: Extreme Case Analysis 

This subproblem aims at identifying the epistemic realizations that prescribes the extreme 
values of J\ and J 2 . In particular, the respondents were challenged to do the following tasks. 

Dl) Find the epistemic realizations of the category II and III parameters that yield the 
smallest and the largest value of J\ for the original uncertainty model used in Cl. Do 
the same for the reduced uncertainty model used in C4. 

D2) Find the epistemic realizations of the category II and III parameters that yield the 
smallest and the largest value of J 2 for the original uncertainty model used in C2. Do 
the same for the reduced uncertainty model used in C4. 
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D3) Identify a few representative realizations of x leading to J 2 > 0. These realizations 
should typify different failure scenarios. Describe the corresponding relationships be- 
tween x and g qualitatively, e.g., the combination of small values of aq and large values 
of x 2 yields to violations in the g 1 < 0 and g± < 0 requirements. 

The response to subproblems C and D should be in agreement. Note however, that some 
approaches for addressing subproblem C are unable to find the extreme-case epistemic real- 
izations sought for here. 

Subproblem E: Robust Design 

In this section we consider the multidisciplinary system having the uncertain parameter 
p G M 21 and the design variable d G M 14 as inputs; and the same g G M 8 used previously 
as output. The objective of this subproblem is to identify design points d with improved 
robustness/reliability characteristics. In particular, the respondents were challenged to do 
the following tasks. 

El) Find a design point d that minimizes largest value of J\ for the uncertainty model used 
in task C4. Provide the resulting value of d and the corresponding range of expected 
values. In regard to the range of Ji, is the resulting design better than d base i ine ? 

E2) Find a design point d that minimizes the largest value of J 2 for the uncertainty model 
used in task C4. Provide the resulting value of d and the corresponding range of failure 
probabilities. In regard to the range of J 2 , determine if the resulting design is better 

than dbaseline? 

E3) Apply the sensitivity analysis of task B2 to the design point found in El, and the 
sensitivity analysis of task B3 to the design point found in E2. Compare the rankings 
with those for the baseline design performed previously. 

IV. The Physical System 

The mathematical model S describes the dynamics of the Generic Transport Model 
(GTM), a remotely operated twin-jet aircraft developed by NASA Langley Research Center. 
Figure 1 shows the flight test article and its concept of operations. The aircraft is piloted 
from a ground station via radio frequency links by using on-board cameras and synthetic 
vision technology. The parameters in p are used to describe losses in control effectiveness and 
time delays resulting from telemetry and communications as well as to model a spectrum 
of flying conditions that extend beyond the normal flying envelope. The requirements in 
g are used to describe the vehicles stability and performance characteristics in regard to 
pilot command tracking and handling/riding qualities. Additional information is available 
in [4, 5, 6, 7]. The “black box” format of the formulation of this challenge problem aims at 
making the problem amenable to the largest possible audience without favoring or hindering 
respondents depending upon their particular field of expertise. 
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Figure 1. NASA GTM test article and its concept of operations. 


V. Software 

The MATLAB® scripts associated with all five subproblems can be downloaded from 
http://uqtools.larc.nasa.gov/nda-uq-challenge-problem-2014/ ■ These scripts require the Con- 
trol System Toolbox to run. The m-files and datafiles required to study each of the subprob- 
lems are as follows: 

• Subproblem A: p_to_xl.m, xlsamplesl .mat and xlsamples2 .mat. 

• Subproblems B, C, and D: p_to_x.m and x_to_g.m. 

• Subproblem E: p_to_x.m and x_and_d_to_g .m. 

The function montecarlo .m, which is not associated with any subproblem in particular, 
carries out a Monte Carlo simulation of g. Details on the inputs and outputs of these scripts 
can be found by looking at internal comments. 
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